Skip to content

fix(config): stop flagging plugin toolsets as unknown in platform_toolsets validation - #97374

Closed
chrisluersen wants to merge 2 commits into
NousResearch:mainfrom
chrisluersen:fix/toolset-validation-plugin-toolsets
Closed

chrisluersen wants to merge 2 commits into
NousResearch:mainfrom
chrisluersen:fix/toolset-validation-plugin-toolsets

Conversation

@chrisluersen

Copy link
Copy Markdown

Summary

platform_toolsets validation (added for #38798) flags plugin-registered toolsets as unknown on every startup, even though they resolve fine at runtime:

⚠️  platform 'cli' references unknown toolset 'a2a' — did you mean 'hermes-cli'?
⚠️  platform 'cli' references unknown toolset 'buzz' — did you mean 'hermes-cli'?
⚠️  platform 'cli' references unknown toolset 'eikon' — did you mean 'hermes-cli'?
⚠️  platform 'discord' references unknown toolset 'eikon' — did you mean 'hermes-discord'?

Root cause

Plugin toolsets (eikon, buzz, a2a) register in the tool registry only after plugins load — which is after migrate_config() runs its platform_toolsets validation. So validate_toolset() genuinely doesn't know them yet at validation time, and the warning is a false positive. The tools are not actually broken — _get_platform_tools() correctly recognizes plugin toolsets at runtime (see #81163).

Fix

The validator now accepts an optional plugin_toolset_names set (the union of known_plugin_toolsets from config — the per-platform plugin-toolset set persisted by the hermes tools save flow). Names carried there validate clean, while genuinely unknown/corrupted names still warn exactly as before.

Test plan

  • New unit tests: plugin-registered toolsets pass without warnings; real corruption (hermes) is still flagged even when plugin names are supplied.
  • tests/hermes_cli/test_toolset_validation.py, test_tools_config.py, test_config_set_list_values.py — 61 passed, 6 skipped.
  • E2E: ran migrate_config() against a real config containing a2a/buzz/eikon — zero toolset warnings emitted.

Fixes the false-positive half of #38798's validation without weakening its corruption detection.

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard area/config Config system, migrations, profiles labels Aug 28, 2026
@Enough1122

Copy link
Copy Markdown

AI code review — automated review for reference; please use your judgment.

Overall: Stops false "unknown toolset" warnings for plugin toolsets that register after config validation (#81163).

What it does

  • hermes_cli/toolset_validation.py:validate_platform_toolsets gains plugin_toolset_names: object = None param; skips any toolset name in that set.
  • hermes_cli/config.py:migrate_config now reads known_plugin_toolsets from raw config (persisted per-platform plugin-toolset set written by hermes tools save flow) via read_raw_config(), flattens string or list values into _plugin_names: set, passes to validator. Handles shapes str vs list vs missing.

Non-blocking notes

  • Covers eikon/buzz/a2a plugin toolsets that exist in registry only after plugin load — correct reasoning that validate_toolset alone flagged them unknown.
  • Flattening with isinstance(_names, str) vs list handles legacy single-string entries — robust.

Non-blocking — please use your judgment.

…lsets validation

Plugin-registered toolsets (eikon, buzz, a2a) only enter the tool registry
after plugins load — which is after config-time validation runs in
migrate_config(). validate_toolset() therefore flags them as unknown even
though they resolve fine at runtime, spamming every startup with:

  platform 'cli' references unknown toolset 'a2a' — did you mean 'hermes-cli'?

Treat names recorded in known_plugin_toolsets (the per-platform plugin-toolset
set persisted by the `hermes tools` save flow) as valid too. The NousResearch#38798
corruption check still fires for genuinely unknown names.
…m masking)

The prior fix collected the union of all platforms' known_plugin_toolsets
and treated every name as valid for every platform, which silently masked
a genuinely wrong per-platform entry (live config: 'a2a' on cli, only
known for discord). validate_platform_toolsets now accepts a per-platform
mapping and resolves plugin names against the current platform only,
while keeping flat-iterable backward compat for existing callers.

Adds regression tests: per-platform names do not cross-mask, and the
mapping shape still accepts known plugins.
@chrisluersen
chrisluersen force-pushed the fix/toolset-validation-plugin-toolsets branch from 3cd4cd5 to 05b2277 Compare August 29, 2026 19:08
@rhein1

rhein1 commented Sep 8, 2026

Copy link
Copy Markdown

Live confirmation from a Hermes v0.21.1 deployment: config migration false-flagged the configured a2a plugin toolset before plugin discovery, while the same toolset resolved and served correctly at runtime. We carried the per-platform config-validation equivalent locally; focused toolset/config tests passed 25/25 and the post-restart A2A card was healthy. The per-platform scoping in this PR is important so one platform's plugin names cannot mask another platform's typo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants